When video is streamed over the Internet and played back through a Web browser or media player, the video is delivered in digital form. Digital video is also used when video is delivered through many broadcast services, satellite services and cable television services. Real-time videoconferencing often uses digital video, and digital video is used during video capture with most smartphones, Web cameras and other video capture devices.
Digital video can consume an extremely high amount of bits. The number of bits that is used per second of represented video content is known as the bit rate. Engineers use compression (also called source coding or source encoding) to reduce the bit rate of digital video. Compression decreases the cost of storing and transmitting video information by converting the information into a lower bit rate form. Decompression (also called decoding) reconstructs a version of the original information from the compressed form. A “codec” is an encoder/decoder system.
Over the last 25 years, various video codec standards have been adopted, including the ITU-T H.261, H.262 (MPEG-2 or ISO/IEC 13818-2), H.263, H.264 (MPEG-4 AVC or ISO/IEC 14496-10), and H.265 (ISO/IEC 23008-2) standards, the MPEG-1 (ISO/IEC 11172-2) and MPEG-4 Visual (ISO/IEC 14496-2) standards, and the SMPTE 421M (VC-1) standard. A video codec standard typically defines options for the syntax of an encoded video bitstream, detailing parameters in the bitstream when particular features are used in encoding and decoding. In many cases, a video codec standard also provides details about the decoding operations a video decoder should perform to achieve conforming results in decoding. Aside from codec standards, various proprietary codec formats define other options for the syntax of an encoded video bitstream and corresponding decoding operations.
Conventionally, a video playback tool includes a video decoder as well as functionality to render reconstructed video pictures for display. Video decoding and rendering operations can be computationally intensive. In particular, while some video decoding operations are relatively simple, others are computationally complex. For example, inverse frequency transforms, fractional sample interpolation operations for motion compensation, in-loop deblock filtering, post-processing filtering, color conversion, and video re-sizing can require extensive computation. This computational complexity can be problematic in various scenarios, such as decoding of high-quality, high-bit rate video (e.g., compressed high-definition video) or decoding of low-latency video (e.g., for remote desktop conferencing or real-time communication).
Thus, some video playback tools use hardware acceleration to offload certain computationally intensive operations to a graphics processor or other special-purpose hardware. For example, in some configurations, a computer system includes a primary central processing unit (“CPU”) (or multiple primary CPUs) as well as a graphics processing unit (“GPU”) or other hardware specially adapted for graphics processing or video decoding. A video playback tool uses the primary CPU as a host to control overall decoding and uses the GPU (or special-purpose decoding hardware) to perform operations that collectively require extensive computation, accomplishing video acceleration. In a typical software architecture for hardware-accelerated video decoding, a host controls overall decoding and may perform some operations such as bitstream parsing using the CPU(s). The decoder signals data to a device driver for an accelerator (e.g., with a GPU) across an acceleration interface.
A video playback tool often requests encoded video from a server over a network. For this reason, a video playback tool may be referred to as a “client.” A video playback tool can be adapted to the computing platform on which the video playback tool runs (“native client”). For example, a native client can be specially adapted to run on a particular operating system (“OS”) and/or hardware configuration, using libraries of the OS or using operations specific to a CPU. When the hardware configuration includes a GPU, the native client can use the GPU to perform certain decoding and rendering operations, which can further speed up the decoding and rendering process. On the other hand, the process of developing native clients for different OSs, platforms, etc. can be time-consuming, especially as the number of versions of OSs and hardware configurations increases, and the installation process for native clients can be complicated.
According to an alternative approach, a video playback tool manages video playback in conjunction with a Web browser. The browser-based video playback tool can include routines in a scripting language or other browser-executable programming language, which can be run in a Web browser environment on diverse computing platforms and OSs. The routines are specified in a platform-agnostic way, which simplifies the development process and installation process. On the other hand, previous browser-based video playback tools can be overwhelmed by the computational complexity of video decoding and rendering operations, especially for time-sensitive scenarios such as remote desktop presentation and real-time communication.